Software Effort Prediction Using Regression Rule Extraction from Neural Networks.

Neural Network Rule Extraction and the LED Display Recognition Problem.

Understanding consumer heterogeneity A business intelligence application of neural networks.

A note on knowledge discovery using neural networks and its application to credit card screening.

Predicting consumer preference for fast%2Dfood franchises a data mining approach.

Market research applications of artificial neural networks.

Rule Extraction from Support Vector Machines An Overview of Issues and Application in Credit Scoring.

Improved SOM Labeling Methodology for Data Mining Applications.

Recursive Neural Network Rule Extraction for Data With Mixed Attributes.

Minerva Sequential Covering for Rule Extraction.

Greedy rule generation from discrete data and its use in neural network rule extraction.

Greedy rule generation from discrete data and its use in neural network rule extraction.

From Knowledge Discovery to Implementation A Business Intelligence Approach Using Neural Network Rule Extraction and Decision Tables.

Separating core and noncore knowledge an application of neural network rule extraction to a cross%2Dnational study of brand image perception.

Extracting salient dimensions for automatic SOM labeling.

A Hybrid SOM%2DSVM Method for Analyzing Zebra Fish Gene Expression.

Applying the Conjugate Gradient Method for Text Document Categorization.

An approach to generate rules from neural networks for regression problems.



Visualizing Globalization A Self%2DOrganizing Maps Approach to Customer Profiling.

Using Neural Network Rule Extraction and Decision Tables for Credit %2D Risk Evaluation.

A Comparative Study of Centroid%2DBased

Effective Query Size Estimation Using Neural Networks.

Generating Concise Sets of Linear Regression Rules from Artificial Neural Networks.

Input Selection in Data%2DDriven Fuzzy Modelling.
An iterative backward selection method for determination of relevant input variables in data-driven fuzzy modeling is presented. The method utilizes parameters of the Takagi-Sugeno model as a factor to determine the significance of input variables. As a result, it is less computationally intensive than most of the existing methods for input variable selection. I. 
Building Credit%2DRisk Evaluation Expert Systems Using Neural Network Rule Extraction and Decision Tables.

An Effective Method for Generating Multiple Linear Regression Rules from Artificial Neural Networks.

Generating Linear Regression Rules from Neural Networks Using Local Least Squares Approximation.

Feedforward Neural Network Construction Using Cross Validation.
The problem of determining the architecture of a multilayer perceptron together with the disadvantages of the standard backpropagation algorithm, directed the research towards algorithms that determine not only the weights but also the structure of the network necessary for learning the data. We propose a Constructive Algorithm with Multiple Operators using Statistical Test (MOST) for determining the architecture. The networks that are constructed by MOST can have multiple hidden layers with multiple hidden units in each layer. The algorithm uses node removal, addition and layer addition and determines the number of nodes in layers by heuristics. It applies a statistical test to compare different architectures. The results are promising and near optimal.
Feedforward Neural Network Construction Using Cross Validation.
The problem of determining the architecture of a multilayer perceptron together with the disadvantages of the standard backpropagation algorithm, directed the research towards algorithms that determine not only the weights but also the structure of the network necessary for learning the data. We propose a Constructive Algorithm with Multiple Operators using Statistical Test (MOST) for determining the architecture. The networks that are constructed by MOST can have multiple hidden layers with multiple hidden units in each layer. The algorithm uses node removal, addition and layer addition and determines the number of nodes in layers by heuristics. It applies a statistical test to compare different architectures. The results are promising and near optimal.
Rapid 3D Model Acquisition from Images of Small Objects.

Opening the neural network black box an algorithm for extracting rules from function approximating artificial neural networks.

Neural Network Pruning for Function Approximation.
Neural networks have been successfully applied to solve a variety of application problems including classification and function approximation. They are especially useful as function approximators because they do not require prior knowledge of the input data distribution and they have been shown to be universal approximators. In many applications, it is desirable to extract knowledge that can explain how the problems are solved by the networks. Most existing approaches have focused on extracting symbolic rules for classification. Few methods have been devised to extract rules from trained neural networks for regression. This article presents an approach for extracting rules from trained neural networks for regression. Each rule in the extracted rule set corresponds to a subregion of the input space and a linear function involving the relevant input attributes of the data approximates the network output for all data samples in this subregion. Extensive experimental results on 32 benchmark data sets demonstrate the effectiveness of the proposed approach in generating accurate regression rules.
Pruned Neural Networks for Regression.
Neural networks have been widely used as a tool for regression. They are capable of approximating any function and they do not require any assumption about the distribution of the data. The most commonly used architectures for regression are the feedforward neural networks with one or more hidden layers. In this paper, we present a network pruning algorithm which determines the number of units in the input and hidden layers of the networks. We compare the performance of the pruned networks to four regression methods namely, linear regression (LR), Naive Bayes (NB), k-nearest-neighbor (kNN), and a decision tree predictor M5 0 . On 32 publicly available data sets tested, the neural network method outperforms NB and kNN if the prediction errors are computed in terms of the root mean squared errors. Under this measurement metric, it also performs as well as LR and M5 0 . On the other hand, using the mean absolute error as the measurement metric, the neural network metho...
FERNN An Algorithm for Fast Extraction of Rules from Neural Networks.
Before symbolic rules are extracted from a trained neural network, the network is usually pruned so as to obtain more concise rules. Typical pruning algorithms require retraining the network which incurs additional cost. This paper presents FERNN, a fast method for extracting rules from trained neural networks without network retraining. Given a fully connected trained feedforward network with a single hidden layer, FERNN first identifies the relevant hidden units by computing their information gains. For each relevant hidden unit, its activation values is divided into two subintervals such that the information gain is maximized. FERNN finds the set of relevant network connections from the input units to this hidden unit by checking the magnitudes of their weights. The connections with large weights are identified as relevant. Finally, FERNN generates rules that distinguish the two subintervals of the hidden activation values in terms of the network inputs. Experimental results show th...
Generating concise and accurate classification rules for breast cancer diagnosis.
In our previous work, we have presented an algorithm that extracts classification rules from trained neural networks and discussed its application to breast cancer diagnosis. In this paper, we describe how the accuracy of the networks and the accuracy of the rules extracted from them can be improved by a simple pre-processing of the data. Data pre-processing involves selecting the relevant input attributes and removing those samples with missing attribute values. The rules generated by our neural network rule extraction algorithm are more concise and accurate than those generated by other rule generating methods reported in the literature. Keywords. Neural network rule extraction; Wisconsin breast cancer diagnosis; Data pre-processing; Attribute selection.  1 Introduction  Neural networks have been shown to be valuable tools for pattern classification in various areas of the medical field [2, 3, 5, 6, 7, 9, 10, 12, 20, 21]. Given an input pattern, the neural network prediction is norm...
A comparison between two neural network rule extraction techniques for the diagnosis of hepatobiliary disorders.
Neural networks have been widely used as tools for prediction in medicine. We  expect to see even more applications of neural networks for medical diagnosis as recently  developed neural network rule extraction algorithms make it possible for the  decision process of a trained network to be expressed as classification rules. These  rules are more comprehensible to a human user than the classification process of the  networks which involves complex nonlinear mapping of the input data. This paper  reports the results from two neural network rule extraction techniques, NeuroLinear  and NeuroRule applied to the diagnosis of hepatobiliary disorders. The data set consists  of nine measurements collected from patients in a Japanese hospital and these  measurements have continuous values. NeuroLinear generates piece-wise linear discriminant  functions for this data set. The continuous measurements have previously  been discretized by domain experts. NeuroRule is applied to the discretized data...


A connectionist approach to generating oblique decision trees.

On mapping decision trees and neural networks.
There exist several methods for transforming decision trees to neural networks. These methods typically construct the networks by directly mapping decision nodes or rules to the neural units. As a result, the networks constructed are often larger than necessary. This paper describes a pruning-based method for mapping decision trees to neural networks, which can compress the network by removing unimportant and redundant units and connections. In addition, equivalent decision trees extracted from the pruned networks are simpler than those induced by well-known algorithms such as ID3 and C4.5. Keywords: decision trees, neural networks, pruning. 1 Introduction Decision trees have been widely used for nonparametric pattern classification tasks which involve several pattern classes and a large number of features. Given an input pattern, the tree classifier performs the comparisons stipulated in each decision node of the tree, and then branches to either the left or the right subtree based on...
Explanation of the %22virtual input%22 phenomenon.
 is obvious that the activation of the corner units are synchronized; likewise for the side units. As a result, the trained network can only produce the 8 training output  8 {0,0,0} 0 {0,0,1} {0,1,0} {1,0,0} {1,0,1} {1,1,0} 1 2 5 6 3 {1,1,1} {0,1,1} 4 7 Figure 1: The eight training samples (0--7) consists of 3-bit inputs and 3 \Theta 3 output patterns. Pattern 8 is a novel pattern generated by the trained network when its connections are pruned. patterns depending on which hidden units are activated. Even if the hidden layer contains real-numbered instead of binary units, the network will still generate the trained output patterns. Furthermore, even if the connection weights are not exactly equal to the nominal values of \Gamma2, 0, and +2, the network will still generate the trained output patterns as long as the weights are close to the nominal values. The closer are the weights to the nominal values, the higher is the frequency of gen
Feature Transformation and Multivariate Decision Tree Induction.
. Univariate decision trees (UDT&#039;s) have inherent problems  of replication, repetition, and fragmentation. Multivariate decision trees  (MDT&#039;s) have been proposed to overcome some of the problems. Close  examination of the conventional ways of building MDT&#039;s, however, reveals  that the fragmentation problem still persists. A novel approach  is suggested to minimize the fragmentation problem by separating hyperplane  search from decision tree building. This is achieved by feature  transformation. Let the initial feature vector be x, the new feature vector  after feature transformation T is y, i.e., y = T (x). We can obtain  an MDT by (1) building a UDT on y; and (2) replacing new features y  at each node with the combinations of initial features x. We elaborate  on the advantages of this approach, the details of T , and why it is expected  to perform well. Experiments are conducted in order to confirm  the analysis, and results are compared to those of C4.5, OC1, and CART.  1 Introducti...
Incremental Feature Selection.
Feature selection is a problem of finding relevant features. When the  number of features of a dataset is large and its number of patterns is huge,  an effective method of feature selection can help in dimensionality reduction.  An incremental probabilistic algorithm is designed and implemented  as an alternative to the exhaustive and heuristic approaches. Theoretical  analysis is given to support the idea of the probabilistic algorithm in  finding an optimal or near-optimal subset of features. Experimental results  suggest that (1) the probabilistic algorithm is effective in obtaining  optimal/suboptimal feature subsets; (2) its incremental version expedites  feature selection further when the number of patterns is large and can  scale up without sacrificing the quality of selected features.  Keywords: Pattern Recognition, Machine Learning, Feature Selection,  Dimensionality Reduction  To appear in Applied Intelligence, Kluwer Academic Publishers.  0  1 Introduction  Feature selection...
Analysis of Hidden Representations by Greedy Clustering.

NeuroLinear A System for Extracting Oblique Decision Rules from Neural Networks.
We present NeuroLinear, a system for extracting oblique decision rules from neural  networks that have been trained for classification of patterns. Each condition of an  oblique decision rule corresponds to a partition of the attribute space by a hyperplane  that is not necessarily axis-parallel. Allowing a set of such hyperplanes to form the  boundaries of the decision regions leads to a significant reduction in the number of  rules generated while maintaining the accuracy rates of the networks. We describe the  components of NeuroLinear in detail by way of two examples using artificial datasets.  Our experimental results on real-world datasets show that the system is effective in  extracting compact and comprehensible rules with high predictive accuracy from neural  networks.  Keywords: Rule extraction, oblique-rule, pruning, discretization.  1 Introduction  Neural networks have been widely applied to solve classification problems. Comparisons between neural networks and decision tre...
Feature Selection via Discretization.
Discretization can turn numeric attributes into discrete ones. Feature selection can eliminate some irrelevant and/or redundant attributes. Chi2 is a simple and general algorithm that uses the   2  statistic to discretize numeric attributes repeatedly until some inconsistencies are found in the data. It achieves feature selection via discretization. It can handle mixed attributes, work with multiclass  data, and remove irrelevant and redundant attributes.  Keywords--- discretization, feature selection, pattern classification I. Introduction  Feature selection can eliminate some irrelevant and/or redundant attributes. By using relevant features, classification algorithms can in general improve their predictive accuracy, shorten the learning period, and form simpler concepts. There are abundant feature selection algorithms. Some use methods like principle component to compose a smaller number of new features [11,12]; some select a subset of the original attributes [1,5]. This paper consi...
A Penalty%2DFunction Approach for Pruning Feedforward Neural Networks.
This paper proposes the use of a penalty function for pruning feedforward neural  network by weight elimination. The penalty function proposed consists of two terms;  the first term is to discourage the use of unnecessary connections and the second term  is to prevent the weights of the connections from taking excessively large values. Simple  criteria for eliminating weights from the network are also given. The effectiveness of this  penalty function is tested on three well known problems. These test problems are the  contiguity problem, the parity problems, and the monks problems. The resulting pruned  networks obtained for many of these problems have fewer connections than previously  reported in the literature.  1 Introduction  We are concerned in this paper with finding a minimal feedforward backpropagation neural network for solving the problem of distinguishing patterns from two or more sets in n-dimensional space. Backpropagation feedforward neural networks have been gaining ac...
Extracting Rules from Neural Networks by Pruning and Hidden%2DUnit Splitting.
An algorithm for extracting rules from a standard three-layer feedforward neural  network is proposed. The trained network is first pruned not only to remove redundant  connections in the network, but more importantly, to detect the relevant inputs. The  algorithm generates rules from the pruned network by considering only a small number  of activation values at the hidden units. If the number of inputs connected to a hidden  unit is sufficiently small, then rules that describe how each of its activation values is  obtained can be readily generated. Otherwise, the hidden unit will be split and treated  as output units, with each output unit corresponding to an activation value. A hidden  layer is inserted and a new subnetwork is formed, trained, and pruned. This process is  repeated until every hidden unit in the network has a relatively small number of input  units connected to it. Examples on how the proposed algorithm works are shown using  real-world data arising from molecular bio...
On the solution of the parity problem by a single hidden layer feedforward neural network.
It is known that the N-bit parity problem is solvable by a standard feedforward neural network having a single hidden layer consisting of (N/2) + 1 hidden units if N is even and (N+1)/2 hidden units if N is odd. The network does not allow a direct connection between the input layer and the output layer and the transfer function used in all hidden units and the output unit is the usual sigmoidal function oe(x) = 1=(1 + exp(\Gammax)). We show that such a solution can be easily obtained by solving a system of linear equations. Keywords: Parity problem, feedforward network, sigmoid function. 1. Introduction The parity problem is one of the more widely used problems for testing neural network training algorithms. This problem is a mapping problem where the domain set consists of all distinct N-bit binary vectors and the result of the mapping is 0 if the number of ones in the vector is even, and 1 otherwise. The problem is considered to be very hard since the output changes whenever any sing...
NeuroLinear From neural networks to oblique decision rules.
We present NeuroLinear, a system for extracting oblique decision rules from neural  networks that have been trained for classification of patterns. Each condition of an  oblique decision rule corresponds to a partition of the attribute space by a hyperplane  that is not necessarily axis-parallel. Allowing a set of such hyperplanes to form the  boundaries of the decision regions leads to a significant reduction in the number of  rules generated while maintaining the accuracy rates of the networks. We describe the  components of NeuroLinear in detail by way of two examples using artificial datasets.  Our experimental results on real-world datasets show that the system is effective in  extracting compact and comprehensible rules with high predictive accuracy from neural  networks.  Keywords: Rule extraction, oblique-rule, pruning, discretization.  1 Introduction  Neural networks have been widely applied to solve classification problems. Comparisons between neural networks and decision tre...
A Probabilistic Approach to Feature Selection %2D A Filter Solution.
Feature selection can be defined as a problem  of finding a minimum set of M relevant attributes  that describes the dataset as well as  the original N attributes do, where M  N .  After examining the problems with both the  exhaustive and the heuristic approach to feature  selection, this paper proposes a probabilistic  approach. The theoretic analysis and  the experimental study show that the proposed  approach is simple to implement and  guaranteed to find the optimal if resources  permit. It is also fast in obtaining results  and effective in selecting features that improve  the performance of a learning algorithm.  An on-site application involving huge  datasets has been conducted independently.  It proves the effectiveness and scalability of  the proposed algorithm. Discussed also are  various aspects and applications of this feature  selection algorithm.  1 Introduction  The problem of feature selection can be defined as finding  M relevant attributes among the N original attrib...
Feature Selection and Classification %2D A Probabilistic Wrapper Approach.
Feature selection is defined as a problem to find a minimum set of M features for an inductive algorithm to achieve the highest predictive accuracy from the data described by the original N features where M  N . A probabilistic wrapper model is proposed as another method besides the exhaustive search and the heuristic approach. The aim of this model is to avoid local minima and exhaustive search. The highest predictive accuracy is the criterion in search of the smallest M . Analysis and experiments show that this model can effectively find relevant features and remove irrelevant ones in the context of improving the predictive accuracy of an induction algorithm. It is simple, straightforward, and providing fast solutions while searching for the optimal. The applications of such a model, its future work and some related issues are also discussed. 1. INTRODUCTION  The problem of feature selection can be defined as finding relevant M features among the original N  attributes, where M  N , ...
Improving Backpropagation Learning with Feature Selection.
There exist redundant, irrelevant and noisy data. Using proper data to train a network  can speed up training, simplify the learned structure, and improve its performance. A  two-phase training algorithm is proposed. In the first phase, the number of input units of  the network is determined by using an information base method. Only those attributes  that meet certain criteria for inclusion will be considered as the input to the network. In  the second phase, the number of hidden units of the network is selected automatically  based on the performance of the network on the training data. One hidden unit is added  at a time only if it is necessary. The experimental results show that this new algorithm  can achieve a faster learning time, a simpler network and an improved performance.  
Improving Backpropagation Learning with Feature Selection.
There exist redundant, irrelevant and noisy data. Using proper data to train a network  can speed up training, simplify the learned structure, and improve its performance. A  two-phase training algorithm is proposed. In the first phase, the number of input units of  the network is determined by using an information base method. Only those attributes  that meet certain criteria for inclusion will be considered as the input to the network. In  the second phase, the number of hidden units of the network is selected automatically  based on the performance of the network on the training data. One hidden unit is added  at a time only if it is necessary. The experimental results show that this new algorithm  can achieve a faster learning time, a simpler network and an improved performance.  
Extracting rules from pruned networks for breast cancer diagnosis.

Extracting rules from pruned networks for breast cancer diagnosis.

Symbolic Representation of Neural Networks.
Although backpropagation neural networks generally predict better than decision trees  do for pattern classification problems, they are often regarded as black boxes, i.e., their  predictions cannot be explained as those of decision trees. In many applications, more  often than not, explicit knowledge is needed by human experts. This work derives symbolic  representations from a neural network to make epxlicit each prediction of the network. An  algorithm is proposed and implemented to extract symbolic rules from neural networks.
Effective Data Mining Using Neural Networks.
Classification is one of the data mining problems receiving great attention recently in the database community.  This paper presents an approach to discover symbolic classification rules using neural networks. Neural networks  have not been thought suited for data mining because how the classifications were made is not explicitly stated  as symbolic rules that are suitable for verification or interpretation by humans. With the proposed approach,  concise symbolic rules with high accuracy can be extracted from a neural network. The network is first trained  to achieve the required accuracy rate. Redundant connections of the network are then removed by a network  pruning algorithm. The activation values of the hidden units in the network are analyzed, and classification rules  are generated using the result of this analysis. The effectiveness of the proposed approach is clearly demonstrated  by the experimental results on a set of standard data mining test problems.  1 Introduction  One ...
Dimensionality reduction via discretization.
The existence of numeric data and large amounts of records in a database  pose a challenging task to explicit concepts extraction from the raw data. This  paper introduces a method that reduces data vertically and horizontally, keeps  the discriminating power of the original data, and paves the way for extracting  concepts. The method is based on discretization (vertical reduction) and feature  selection (horizontal reduction). The experimental results show that (1) the data  can be effectively reduced by the proposed method; (2) the predictive accuracy of  a classifier (C4.5) can be improved after data and dimensionality reduction; and  (3) the classification rules learned are simpler.  Key Word: Dimensionality Reduction, Discretization, Knowledge Discovery  0  1 Introduction  The wide use of computers brings forth proliferation of databases. Without the aid of computer, little of this flood of raw data will ever be seen and exploited by humans. Knowledge discovery systems in database...
Understanding Neural Networks via Rule Extraction.
Although backpropagation neural networks  generally predict better than decision trees do  for pattern classification problems, they are often  regarded as black boxes, i.e., their predictions  are not as interpretable as those of decision  trees. This paper argues that this is because  there has been no proper technique that  enables us to do so. With an algorithm that  can extract rules, by drawing parallels with  those of decision trees, we show that the predictions of a network can be explained via rules extracted  from it, thereby, the network can be understood. Experiments demonstrate that rules  extracted from neural networks are comparable  with those of decision trees in terms of predictive  accuracy, number of rules and average  number of conditions for a rule; they preserve  high predictive accuracy of original networks. 
Towards Effective Classfication Rule Extraction %28Abstract%29.

NeuroRule A Connectionist Approach to Data Mining.
Classification, which involves finding rules  that partition a given data set into disjoint  groups, is one class of data mining problems.  Approaches proposed so far for mining classification  rules for large databases are mainly decision  tree based symbolic learning methods.  The connectionist approach based on neural  networks has been thought not well suited  for data mining. One of the major reasons  cited is that knowledge generated by neural  networks is not explicitly represented in the  form of rules suitable for verification or interpretation  by humans. This paper examines  this issue. With our newly developed algorithms,  rules which are similar to, or more  concise than those generated by the symbolic  methods can be extracted from the neural networks.  The data mining process using neural  networks with the emphasis on rule extraction  is described. Experimental results and comparison  with previously published works are  presented. 
A Neural Network Construction Algorithm which Maximizes the Likelihood Function.

Efficient Neural Network Training on a Cray y%2DMP.

A Neural Network Construction Algorithm with Application to Image.

